Design

Assignment schemes

Macartan Humphreys

1 Design

1.1 Topics

  • Focus on randomization schemes

1.1.1 Experiments

  • Experiments are investigations in which an intervention, in all its essential elements, is under the control of the investigator. (Cox & Reid)

  • Two major types of control:

  1. control over assignment to treatment – this is at the heart of many field experiments
  2. control over the treatment itself – this is at the heart of many lab experiments
  • Main focus today is on 1 and on the question: how does control over assignment to treatment allow you to make reasonable statements about causal effects?

1.1.2 Experiments

1.1.3 Basic randomization

  • Basic randomization is very simple. For example, say you want to assign 5 of 10 units to treatment. Here is simple code:
1:10 %in% sample(1:10, 5)    
 [1] FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE FALSE  TRUE  TRUE

1.1.4 …should be replicable

In general you might want to set things up so that your randomization is replicable. You can do this by setting a seed:

set.seed(20111112)
1:10 %in% sample(1:10, 5) 
 [1] FALSE  TRUE FALSE  TRUE FALSE  TRUE  TRUE FALSE  TRUE FALSE
set.seed(20111112)
1:10 %in% sample(1:10, 5) 
 [1] FALSE  TRUE FALSE  TRUE FALSE  TRUE  TRUE FALSE  TRUE FALSE

1.1.5 Basic randomization

Even better is to set it up so that it can reproduce lots of possible draws so that you can check the propensities for each unit.

set.seed(20111112)
P <- sapply(1:1000, function(i) 1:10 %in% sample(1:10, 5)) 
apply(P, 1, mean)
 [1] 0.495 0.493 0.512 0.522 0.505 0.522 0.479 0.510 0.444 0.518

Here the \(P\) matrix gives 1000 possible ways of allocating 5 of 10 units to treatment. We can then confirm that the average propensity is 0.5.

  • A huge advantage of this approach is that if you make a mess of the random assignment; you can still generate the P matrix and use that for all analyses!

1.1.6 Do it in advance

  • Unless you need them to keep subjects at ease, leave your spinners and your dice and your cards behind.
  • Especially when you have multiple or complex randomizations you are generally much better doing it with a computer in advance

A survey dictionary with results from a complex randomization presented in a simple way for enumerators

1.1.7 Did the randomization ‘’work’’?

  • People often wonder: did randomization work? Common practice is to implement a set of \(t\)-tests to see if there is balance. This makes no sense.

  • If you doubt whether it was implemented properly do an \(F\) test. If you worry about variance specify controls in advance as a function of relation with outcomes (more on this later). If you worry about conditional bias then look at substantive differences between groups, not \(t\)–tests

  • If you want realizations to have particular properties: build it into the scheme in advance.

1.2 Cluster Randomization

1.2.1 Cluster Randomization

  • Simply place units into groups (clusters) and then randomly assign the groups to treatment and control.
  • All units in a given group get the same treatment

Note: clusters are part of your design, not part of the world.

1.2.2 Cluster Randomization

  • Often used if intervention has to function at the cluster level or if outcome defined at the cluster level.

  • Disadvantage: loss of statistical power

  • However: perfectly possible to assign some treatments at cluster level and then other treatments at the individual level

  • Principle: (unless you are worried about spillovers) generally make clusters as small as possible

  • Principle: Surprisingly, variability in cluster size makes analysis harder.

  • Be clear about whether you believe effects are operating at the cluster level or at the individual level. This matters for power calculations.

  • Be clear about whether spillover effects operate only within clusters or also across them. If within only you might be able to interpret treatment as the effect of being in a treated cluster…

1.2.3 Cluster Randomization: Block by cluster size

Surprisingly, if clusters are of different sizes the difference in means estimator is not unbiased, even if all units are assigned to treatment with the same probability.

Here’s the intuition.Say there are two clusters each with homogeneous treatment effects:

Cluster Size Y0 Y1
1 1000000 0 1
2 1 0 0

Then: What is the true average treatment effect? What do you expect to estimate from cluster random assignment?

The solution is to block by cluster size. For more see: http://gking.harvard.edu/files/cluster.pdf

1.3 Blocked assignments and other restricted randomizations

1.3.1 Blocking

There are more or less efficient ways to randomize.

  • Randomization helps ensure good balance on all covariates (observed and unobserved) in expectation.
  • But balance may not be so great in realization
  • Blocking can help ensure balance ex post on observables

Consider a case with four units and two strata. There are 6 possible assignments of 2 units to treatment:

ID X Y(0) Y(1) R1 R2 R3 R4 R5 R6
1 1 0 1 1 1 1 0 0 0
2 1 0 1 1 0 0 1 1 0
3 2 1 2 0 1 0 1 0 1
4 2 1 2 0 0 1 0 1 1
– – – – – – – – – –
\(\widehat{\tau}\): 0 1 1 1 1 2

Even with a constant treatment effect and everything uniform within blocks, there is variance in the estimation of \(\widehat{\tau}\). This can be eliminated by excluding R1 and R6.

1.3.2 Blocking

Simple blocking in R (5 pairs):

sapply(1:5, function(i) rank(runif(2))==1)
1 2 3 4 5
TRUE TRUE TRUE FALSE FALSE
FALSE FALSE FALSE TRUE TRUE

1.3.3 Of blocks and clusters

1.3.4 Blocking

  • Blocking is a case of restricted randomization. Although each unit is sampled with equal probability, the profiles of possible assignments are not.
  • You have to take account of this when doing analysis
  • There are many other approaches.
    • Matched Pairs are a particularly fine approach to blocking
    • You could also randomize and then replace the randomization if you do not like the balance. This sounds tricky (and it is) but it is OK as long as you understand the true lottery process you are employing and incorporate that into analysis
    • It is even possible to block on covariates for which you don’t have data ex ante, by using methods in which you allocate treatment over time as a function of features of your sample (also tricky)

1.3.5 Other types of restricted randomization

  • Really you can set whatever criterion you want for your set of treated units to have (eg no treated unit beside another treated unit; at least 5 from the north, 10 from the south, guaranteed balance by some continuous variable etc)
  • You just have to be sure that you understand the random process that was used and that you can use it in the analysis stage
  • But here be dragons
    • The more complex your design, the more complex your analysis.
    • General injunction http://www.ncbi.nlm.nih.gov/pubmed/15580598 ``as ye randomize so shall ye analyze’’)
    • In general you should make sure that a given randomization procedure coupled with a given estimation procedure will produce an unbiased estimate. DeclareDesign can help with this.

1.3.6 Challenges with re-randomization

  • We can see that blocked and clustered assignments are actually types of restricted randomizations: they limit the set of acceptable randomizations to those with good properties
  • You could therefore implement the equivalent distribution of assignments y specifying an acceptable rule and then re-randomizing when the rule is met
  • That’s fine but you would then have to take account of clustering and blocking just as you do when you actually cluster or block

1.4 Factorial Designs

1.4.1 Factorial Designs

  • Often when you set up an experiment you want to look at more than one treatment.
  • Should you do this or not? How should you use your power?

1.4.2 Factorial Designs

  • Often when you set up an experiment you want to look at more than one treatment.
  • Should you do this or not? How should you use your power?
\(T2=0\) \(T2=1\)
T1 = 0 \(50\%\) \(0\%\)
T1 = 1 \(50\%\) \(0\%\)
\(T2=0\) \(T2=1\)
T1 = 0 \(25\%\) \(25\%\)
T1 = 1 \(25\%\) \(25\%\)
\(T2=0\) \(T2=1\)
T1 = 0 \(33.3\%\) \(33.3\%\)
T1 = 1 \(33.3\%\) \(0\%\)

1.4.3 Factorial Designs

  • Surprisingly adding multiple treatments does not eat into your power (unless you are decomposing a complex treatment – then it can. Why?)
  • Especially when you use a fully crossed design like the middle one above.
  • Fisher: “No aphorism is more frequently repeated in connection with field trials, than that we must ask Nature few questions, or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken.”
  • However – adding multiple treatments does alter the interpretation of your treatment effects. If T2 is an unusual treatment for example, then half the T1 effect is measured for unusual situations.

1.4.4 Factorial Designs: In practice

  • In practice if you have a lot of treatments it can be hard to do full factorial designs – there may be too many combinations.

  • In such cases people use fractional factorial designs, like the one below (5 treatments but only 8 units!)

Variation T1 T2 T3 T4 T5
1 0 0 0 1 1
2 0 0 1 0 0
3 0 1 0 0 1
4 0 1 1 1 0
5 1 0 0 1 0
6 1 0 1 0 1
7 1 1 0 0 0
8 1 1 1 1 1

  • Then randomly assign units to rows. Note columns might also be blocking covariates.

  • In R, look at library(survey)

1.4.5 Factorial Designs: In practice

  • But be careful: you have to be comfortable with possibly not having any simple counterfactual unit for any unit (invoke sparsity-of-effects principle).
Unit T1 T2 T3 T4 T5
1 0 0 0 1 1
2 0 0 1 0 0
3 0 1 0 0 1
4 0 1 1 1 0
5 1 0 0 1 0
6 1 0 1 0 1
7 1 1 0 0 0
8 1 1 1 1 1
  • In R, look at library(survey)

1.4.6 Controversy?

Muralidharan, Romero, and Wüthrich (2023) write:

Factorial designs are widely used to study multiple treatments in one experiment. While t-tests using a fully-saturated “long” model provide valid inferences, “short” model t-tests (that ignore interactions) yield higher power if interactions are zero, but incorrect inferences otherwise. Of 27 factorial experiments published in top-5 journals (2007–2017), 19 use the short model. After including interactions, over half of their results lose significance. […]

1.5 External Validity: Can randomization strategies help?

1.5.1 Principle: Address external validity at the design stage

Anything to be done on randomization to address external validity concerns?

  • Note 1: There is little or nothing about field experiments that makes the external validity problem greater for these than for other ‘’sample based’’ research
  • Note 2: Studies that use up the available universe (cross national studies) actually have a distinct external validity problem
  • Two ways to think about external validity issues:
    1. Are things likely to operate in other units like they operate in these units? (even with the same intervention)
    2. Are the processes in operation in this treatment likely to operate in other treatments? (even in this population)

1.5.2 Principle: Address external validity at the design stage

  • Two ways to think about external validity issues:
    1. Are things likely to operate in other units like they operate in these units? (even with the same intervention) 2.Are the processes in operation in this treatment likely to operate in other treatments? (even in this population)
  • Two approaches for 1.
    • Try to sample cases and estimate population average treatment effects
    • Exploit internal variation: block on features that make the case unusal and assess importance of these (eg is unit poor? assess how effects differ in poor and wealthy components)
  • 2 is harder and requires a sharp identification of context free primitives, if there are such things.

1.6 Assignments with DeclareDesign

1.6.1 A design: Multilevel data

A design with hierarchical data and different assignment schemes.

design <- 
  declare_model(
    school = add_level(N = 16, 
                       u_school = rnorm(N, mean = 0)),     
    classroom = add_level(N = 4,    
                  u_classroom = rnorm(N, mean = 0)),
    student =  add_level(N = 20,    
                         u_student = rnorm(N, mean = 0))
    ) +
  declare_model(
    potential_outcomes(Y ~ .1*Z + u_classroom + u_student + u_school)
    ) +
  declare_assignment(Z = simple_ra(N)) + 
  declare_measurement(Y = reveal_outcomes(Y ~ Z))  +
  declare_inquiry(ATE = mean(Y_Z_1 - Y_Z_0)) + 
  declare_estimator(Y ~ Z, .method = difference_in_means)

1.6.2 Sample data

Here are the first couple of rows and columns of the resulting data frame.

my_data <- draw_data(design)
kable(head(my_data), digits = 2)
school u_school classroom u_classroom student u_student Y_Z_0 Y_Z_1 Z Y
01 -0.77 01 -0.06 0001 0.36 -0.48 -0.38 0 -0.48
01 -0.77 01 -0.06 0002 0.16 -0.67 -0.57 0 -0.67
01 -0.77 01 -0.06 0003 1.04 0.21 0.31 1 0.31
01 -0.77 01 -0.06 0004 1.54 0.70 0.80 0 0.70
01 -0.77 01 -0.06 0005 -0.99 -1.82 -1.72 0 -1.82
01 -0.77 01 -0.06 0006 -0.70 -1.53 -1.43 0 -1.53

1.6.3 Sample data

Here is the distribution between treatment and control:

kable(t(as.matrix(table(my_data$Z))), 
      col.names = c("control", "treatment"))
control treatment
655 625

1.6.4 Complete Random Assignment using the built in function

assignment_complete <-   declare_assignment(Z = complete_ra(N))  

design_complete <- 
  replace_step(design, "assignment", assignment_complete)

1.6.5 Data from complete assignment

We can draw a new set of data and look at the number of subjects in the treatment and control groups.

set.seed(1:5)
data_complete <- draw_data(design_complete)

kable(t(as.matrix(table(data_complete$Z))))
0 1
640 640

1.6.6 Plotted

1.6.7 Block Random Assignment

  • The treatment and control group will in expectation contain the same share of students in different classrooms.
  • But as we saw this does necessarily hold in realization
  • We make this more obvious by sorting the students by treatment status with schools

1.6.8 Blocked design

assignment_blocked <-   
  declare_assignment(Z = block_ra(blocks = classroom))  

estimator_blocked <- 
  declare_estimator(Y ~ Z, blocks = classroom, 
                    .method = difference_in_means)  

design_blocked <- 
  design |> 
  replace_step("assignment", assignment_blocked) |>
  replace_step("estimator", estimator_blocked)

1.6.9 Illustration of blocked assignment

  • Note that subjects are sorted here after the assignment to make it easier to see that in this case blocking ensures that exactly 5 students within each classroom are assigned to treatment.

1.6.10 Clustering

But what if all students in a given class have to be assigned the same treatment?

assignment_clustered <- 
  declare_assignment(Z = cluster_ra(clusters = classroom))  
estimator_clustered <- 
  declare_estimator(Y ~ Z, clusters = classroom, 
                    .method = difference_in_means)  


design_clustered <- 
  design |> 
  replace_step("assignment", assignment_clustered) |> 
  replace_step("estimator", estimator_clustered)

1.6.11 Illustration of clustered assignment

1.6.12 Clustered and Blocked

assignment_clustered_blocked <-   
  declare_assignment(Z = block_and_cluster_ra(blocks = school,
                                              clusters = classroom))  
estimator_clustered_blocked <- 
  declare_estimator(Y ~ Z, blocks = school, clusters = classroom, 
                    .method = difference_in_means)  


design_clustered_blocked <- 
  design |> 
  replace_step("assignment", assignment_clustered_blocked) |> 
  replace_step("estimator", estimator_clustered_blocked)

1.6.13 Illustration of clustered and blocked assignment

1.6.14 Illustration of efficiency gains from blocking

designs <- 
  list(
    simple = design, 
    complete = design_complete, 
    blocked = design_blocked, 
    clustered = design_clustered,  
    clustered_blocked = design_clustered_blocked) 
diagnoses <- diagnose_design(designs)

1.6.15 Illustration of efficiency gains from blocking

Design Power Coverage
simple 0.16 0.95
(0.01) (0.01)
complete 0.20 0.96
(0.01) (0.01)
blocked 0.42 0.95
(0.01) (0.01)
clustered 0.06 0.96
(0.01) (0.01)
clustered_blocked 0.08 0.96
(0.01) (0.01)

1.6.16 Sampling distributions

diagnoses$simulations_df |> 
  mutate(design = factor(design, c("blocked", "complete", "simple", "clustered_blocked", "clustered"))) |>
  ggplot(aes(estimate)) +
  geom_histogram() + facet_grid(~design)

1.6.17 Nasty integer issues

  • In many designs you seek to assign an integer number of subjects to treatment from some set.

  • Sometimes however your assignment targets are not integers.

Example:

  • I have 12 subjects in four blocks of 3 and I want to assign each subject to treatment with a 50% probability.

Two strategies:

  1. I randomly set a target of either 1 or 2 for each block and then do complete assignment in each block. This can result in the numbers treated varying from 4 to 8
  2. I randomly assign a target of 1 for two blocks and 2 for the other 2 blocks

1.6.18 Nasty integer issues

# remotes::install_github("macartan/probra")
library(probra)
set.seed(1)

blocks <- rep(1:4, each = 3)

table(blocks, prob_ra(blocks = blocks))
      
blocks 0 1
     1 1 2
     2 1 2
     3 2 1
     4 2 1
table(blocks, block_ra(blocks = blocks))
      
blocks 0 1
     1 1 2
     2 2 1
     3 1 2
     4 1 2

1.6.19 Nasty integer issues

Can also be used to set targets

# remotes::install_github("macartan/probra")
library(probra)
set.seed(1)

fabricate(N = 4,  size = c(47, 53, 87, 25), n_treated = prob_ra(.5*size)) %>%
  janitor::adorn_totals("row") |> 
  kable(caption = "Setting targets to get 50% targets with minimal variance")
Setting targets to get 50% targets with minimal variance
ID size n_treated
1 47 23
2 53 27
3 87 43
4 25 13
Total 212 106

1.7 Indirect assignments

Indirect control

1.7.1 Indirect assignments

Indirect assignments are generally generated by applying a direct assignment and then figuring our an implied indirect assignment

set.seed(1)

df <-
  fabricate(
    N = 100, 
    latitude = runif(N),
    longitude = runif(N))

adjacency <- 
  sapply(1:nrow(df), function(i) 
    1*((df$latitude[i] - df$latitude)^2 + (df$longitude[i] - df$longitude)^2)^.5 < .1)

diag(adjacency) <- 0

1.7.2 Indirect assignments

adjacency |>  
  reshape2::melt(c("x", "y"), value.name = "z") |> mutate(z = factor(z)) |>
  ggplot(aes(x=x,y=y,fill=z))+
  geom_tile()

1.7.3 Indirect assignments

n_assigned <- 50

design <-
  declare_model(data = df) + 
  declare_assignment(
    direct = complete_ra(N, m = n_assigned),
    indirect = 1*(as.vector(as.vector(direct) %*% adjacency) >= 1))

draw_data(design) |> with(table(direct, indirect))
      indirect
direct  0  1
     0 13 37
     1 13 37

1.7.4 Indirect assignments: Properties

indirect_propensities <- replicate(5000, draw_data(design)$indirect) |> 
  apply(1, mean)

1.7.5 Indirect assignments: Properties

df |> ggplot(aes(latitude, longitude, label = round(indirect_propensities_1, 2))) + geom_text()

1.7.6 Indirect assignments: Redesign

replicate(5000, draw_data(design |> redesign(n_assigned = 25))$indirect) |> 
  apply(1, mean) 
Error in mean(indirect_propensities): object 'indirect_propensities' not found

1.7.7 Indirect assignments: Redesign

df |> ggplot(aes(latitude, longitude, label = round(indirect_propensities_2, 2))) + 
  geom_text()

Looks better: but there are trade offs between the direct and indirect distributions

Figuring out the optimal procedure requires full diagnosis

Muralidharan, Karthik, Mauricio Romero, and Kaspar Wüthrich. 2023. “Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments.” Review of Economics and Statistics, 1–44.